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Abstract 

Acquisition  programs  are  under  pressure  to  deliver  increasingly  complex  capability  to 
the  field  without  the  cost  growth  associated  with  recent  programs.  Evolutionary  acquisition 
was  adopted  to  help  reduce  system  cost  (through  the  use  of  mature  technologies)  and  to 
improve  system  performance  (through  faster  deployment  of  incremental  capability).  While 
the  ultimate  verdict  is  not  yet  in  on  this  decision,  our  previous  simulation-based  results  have 
demonstrated  that  evolutionary  acquisition  can  deliver  improved  capability  more  quickly  than 
traditional  acquisition,  but  that  cost  may  actually  increase  over  that  of  traditional  acquisition. 
This  is  due  to  the  overhead  resulting  from  more  frequent  system  deployment  and  update 
cycles.  Are  there  other  factors  that  can  help  reduce  the  cost  of  evolutionary  acquisition? 
This  paper  investigates  the  role  of  system  modularity  and  production  level  in  the  cost  of 
evolutionary  acquisition.  Modularity  typically  imposes  upfront  costs  in  design  and 
development,  but  may  result  in  downstream  savings  in  production  and  sustainment 
(including  deployment  of  evolutionary  new  capability).  A  simulation  experiment  is  conducted 
to  determine  under  which  conditions  cost  increases  are  minimized. 

Introduction 

In  today's  fiscally  constrained  environment,  cost  is  a  major  issue  that  must  be 
addressed  in  military  acquisition.  In  particular,  cost  growth  is  of  concern,  where  cost  growth 
is  the  amount  by  which  actual  and  projected  costs  increase  over  time  from  earlier  cost 
estimates.  A  recent  report  from  the  Government  Accountability  Office  (GAO)  noted  that  for 
the  fiscal  year  2008  portfolio  of  weapons  systems,  there  has  been  cost  growth  of  $296 
billion  (GAO,  2009).  Cost  growth  can  result  in  fewer  systems  being  produced  than 
envisioned  or  desired  (e.g.,  the  F-22),  or  in  program  cancellation  (e.g.,  the  Navy  Area 
Missile  Defense).  In  the  current  and  projected  fiscal  environment,  there  is  considerable 
pressure  to  reduce  costs  and  to  rein  in  cost  growth. 

One  driver  of  cost  is  the  uncertainty  and  risk  associated  with  development  of  new 
technologies  for  systems.  In  an  effort  to  reduce  this  risk,  and  hence  attempt  to  reduce  costs 
and  cost  growth,  evolutionary  acquisition  was  adopted,  whereby  systems  are  developed 
using  more  mature  technologies  and  deployed  in  increments  of  capability.  At  the  same 
time,  sustainment  cost  is  increasingly  factored  into  the  overall  acquisition  cost  equation, 
especially  as  system  lifetimes  increase.  For  instance,  the  contract  for  the  F-35  Joint  Strike 
Fighter  calls  for  a  combined  production  and  sustainment  supply  network  to  promote  cost 
efficiencies  over  the  system  lifecycle. 
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Not  enough  evidence  has  been  collected  to  establish  the  effectiveness  of 
evolutionary  acquisition  and  preliminary  efforts  to  address  sustainment  costs.  Hence,  our 
work  uses  computer  simulation  to  assess  the  potential  effectiveness  of  different  strategies  to 
address  cost.  Previous  work  indicates  that  evolutionary  acquisition  alone  may  not  be 
sufficient  to  control  costs  (Pennock  &  Rouse,  2008).  This  paper  investigates  the 
effectiveness  of  evolutionary  acquisition  when  system  modularity  and  production  level  are 
considered.  Modularity  is  hypothesized  to  help  reduce  sustainment  costs  associated  with 
maintenance,  repair  and  technology  upgrades. 

The  remainder  of  the  paper  is  organized  as  follows.  Section  2  discusses  acquisition 
costs,  cost  growth  and  the  potential  role  of  system  modularity  in  helping  address  both.  The 
simulation  model  used  to  study  these  issues  is  described  in  Section  3.  Section  4  presents 
an  experiment  conducted  to  test  the  effect  of  modularity  and  production  level  on  cost. 
Section  5  provides  analysis  of  the  overall  experimental  results.  Section  6  focuses  on  those 
results  relevant  to  evolutionary  acquisition.  Finally,  Section  7  concludes  and  presents  future 
research  directions. 

Acquisition  Cost  and  Cost  Growth 

High  cost  and  cost  overruns  have  long  been  an  issue  with  military  systems.  Cost 
growth  occurs  for  a  variety  of  reasons,  including  uncertainty  and  lack  of  knowledge  about 
technology,  design,  and  manufacturing  (GAO,  2009).  Candreva  (2009)  points  to  the  role  of 
institutional  factors  in  organizational  failures  such  as  cost  growth.  One  effort  to  address 
more  cost  effective  acquisition  was  the  introduction  of  evolutionary  acquisition.  Under 
evolutionary  acquisition,  the  focus  is  on  using  technologies  for  new  systems  that  are 
relatively  mature,  as  opposed  to  traditional  acquisition,  which  emphasizes  use  of  new  and 
immature  technologies.  The  theory  is  that  use  of  mature  technologies  tends  to  reduce  cycle 
times  for  new  system  development,  due  to  less  risk  in  the  technology  development  phase  of 
acquisition  (Johnson  &  Johnson,  2002).  This  should  translate  into  reduced  cost  growth. 

Our  previous  research  has  studied  cost  by  focusing  on  the  process  aspects  of 
acquisition.  For  instance,  we  have  demonstrated  that  evolutionary  acquisition  processes 
can  yield  quicker  deployment  of  capability  than  traditional  acquisition  processes,  but  at 
potentially  higher  cost  due  to  overhead  from  the  increased  frequency  of  development  cycles 
(Pennock  &  Rouse,  2008).  This  work  did  not  address  sustainment,  however.  Sustainment 
is  estimated  to  constitute  approximately  60%  of  lifecycle  cost  (Andrews,  2003).  As  a  result, 
the  acquisition  community  is  putting  more  focus  on  sustainment,  its  associated  costs  and  its 
potential  for  cost  growth. 

One  avenue  that  may  help  address  high  cost  in  sustainment  is  the  concept  of  system 
modularity.  Modularity  is  an  important  concept  in  design  of  systems  and  products  (Baldwin 
&  Clark,  2000;  Ulrich  &  Tung,  1991).  Modular  design  seeks  to  reduce  the  dependencies 
between  various  system  components.  This  has  the  potential  to  help  improve  the 
maintainability  of  a  system  over  time  and  to  reduce  the  cost  of  sustainment  by  facilitating 
repair  and  upgrade  activities.  Little  research  has  quantitatively  studied  reduction  in 
sustainment  due  to  modularity,  though.  A  number  of  hypotheses  have  been  formulated  in 
the  research  literature  that  are  of  interest  in  terms  of  the  impact  of  modularity  on  cost. 

Increasing  modularity  decreases  the  cost  of  implementing  technology 
upgrades  for  deployed  systems  (Fleming  &  Sorenson,  2001;  Garud  & 
Kumaraswamy,  1995;  Gershenson,  Prasad  &  Zhang,  2003;  Huang  &  Kusiak, 
1998;  Ulrich,  1995;  Ulrich  &  Tung,  1991). 
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1 .  Increasing  modularity  decreases  the  mean  time  to  repair  a  system 
that  has  failed  and,  hence,  potentially  the  cost  (Cheung  &  Hausman, 
1995;  Gershenson  et  al. ,  2003;  Tsai,  Wang  &  Lo,  2003). 

2.  Increasing  modularity  increases  the  upfront  engineering  design  hours 
required  for  a  system  and  hence  potentially  the  cost  (Ulrich,  1995). 

These  hypotheses  suggest  that  there  is  a  trade-off  between  cost  savings  in 
sustainment  due  to  system  modularity  and  the  cost  of  designing  a  modular  system.  One 
goal  of  this  paper  is  to  explore  this  issue.  Previous  work  has  demonstrated  that  increased 
system  modularity  tends  to  facilitate  reduced  sustainment  costs,  in  terms  of  repair  and 
technology  upgrade  activities  (Bodner,  Smith  &  Rouse,  2009).  The  relationship  is  strongest 
for  high  levels  of  modularity,  with  diminishing  returns  as  modularity  levels  are  reduced. 

Thus,  one  question  is  what  levels  of  modularity  are  required  to  balance  the  upstream  costs 
of  modularity  design  with  the  downstream  savings  from  modularity. 

Model  Description 

The  simulation  model  used  for  this  research  can  be  characterized  as  three 
interacting  sets  of  processes.  First,  there  are  the  systems  being  developed  for  deployment. 
These  are  housed  within  programs  that  conduct  the  various  activities  required  for 
acquisition.  Second,  there  is  the  acquisition  enterprise,  which  consists  of  a  set  of  processes 
through  which  programs  develop  systems.  We  address  both  procurement  and  sustainment. 
Finally,  there  are  exogenous  effects  that  impact  systems  and  acquisition. 

We  use  discrete-event  simulation,  which  has  been  used  extensively  for  analysis  of 
process-oriented  domains  such  as  acquisition  (Law  &  Kelton,  2000).  Our  model  is 
implemented  using  ARENA  12.0,  a  commercially  available  and  widely  used  discrete-event 
simulation  package  (Kelton,  Sadowski  &  Sturrock,  2004). 

System  Model 

The  acquisition  enterprise  develops  a  number  of  different  systems  for  military  use.  In 
our  model,  a  system  is  characterized  primarily  by  its  technologies  in  development  and  by  its 
architecture  in  sustainment.  In  development,  each  system  has  several  technologies  that 
must  be  matured  and  integrated  into  the  system  so  that  the  system  can  be  successfully 
deployed.  Each  technology  has  an  application  area,  a  maturity  level  and  a  capability  level. 
The  application  area  describes  the  function  of  the  technology  (e.g.,  radar  or  stealth).  The 
maturity  level  dictates  its  stage  of  progress  from  new  and  potentially  promising  to  proven 
and  mature.  It  is  measured  using  the  technological  readiness  level  (TRL)  scale  (Kim,  2005) 
recently  adopted  by  the  DoD  (DoD,  2006).  The  capability  level  characterizes  the  functional 
capability  of  the  technology  relative  to  others  in  the  same  application  area. 

In  sustainment,  the  system  architecture  relates  how  different  system  components  are 
arranged  within  the  whole  system.  This  architecture  provides  the  basis  for  systems  to  have 
a  specified  modularity.  In  modeling  system  modularity,  we  assume  that  a  system  consists  of 
n  components,  one  of  which  is  the  system  infrastructure.  The  infrastructure  serves  as  the 
platform  that  integrates  other  components,  and  it  is  assumed  to  be  a  large-scale  and  static 
in  nature  over  the  life  of  the  system  (e.g.,  an  airframe).  Modularity  is  then  conceptualized  as 
a  matrix  denoting  the  relationships  between  components.  A  relationship  exists  when  two 
components  are  connected  with  each  other,  or  more  specifically  when  changes  to  one 
component  necessitate  changes  to  another.  In  complex  systems,  components  may  be 
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organized  into  modules,  which  then  combine  to  form  a  system.  Relationships  can  then  exist 
between  components  within  a  module,  between  modules,  or  between  components  spanning 
two  different  modules.  A  relationship  between  two  components  affects  whether  a  repair  to  a 
particular  component  requires  a  repair  to  another,  and  whether  a  technology  upgrade  to  a 
particular  component  requires  an  upgrade  to  another. 

For  a  system  k,  composed  of  nk  components,  we  define  its  repair  modularity  using  an 
nk  x  nk  matrix  Rk.  Each  entry  r,jk  in  this  matrix  represents  the  degree  to  which  component  /  is 
related  to  component  j  for  purposes  of  repair.  This  is  expressed  as  the  Bernoulli  probability 
that  a  repair  to  component  /  results  in  a  repair  to  component  j.  Similarly,  we  define  U/f  as  the 
modularity  matrix  associated  with  technology  upgrades,  with  uiJk  representing  the  Bernoulli 
probability  that  an  upgrade  to  component  /  requires  a  compatibility  upgrade  to  component  j. 
Note  that  neither  Rk  nor  U/<  is  assumed  to  be  symmetrical.  Figure  1  illustrates  the  concept  of 
a  repair  modularity  matrix  with  nk  =  6.  In  the  matrix,  diagonal  elements  are  defined  to 
equal  1.0.  In  addition,  component  1  is  defined  to  be  the  infrastructure,  and  we  assume  that 
ryk  =1.0  for  all  j  +  1 ,  and  that  rnk  =  0.0  for  all  /  +  1 .  In  other  words,  all  components  are 
assumed  to  be  affected  by  a  change  in  infrastructure,  while  infrastructure  is  assumed  not  to 
be  affected  by  a  change  in  any  component. 
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Figure  1.  System  Modularity 

We  use  the  concept  of  a  modularity  index  to  parameterize  the  extent  to  which  a 
system  is  modular  (Guo  &  Gershenson,  2004;  Holtta-Otto  &  de  Week,  2007).  Since 
modularity  is  matrix-based  and,  hence,  multidimensional  in  nature,  an  index  provides  a  more 
concise  characterization  of  modularity.  Our  particular  index  for  a  repair  modularity  matrix  is 
defined  below: 


1 


,lk  ,lk 

-Y  Y-, 

(k  - 1)(&  -  2)  “2  j=2? 


ijk 


This  index  is  the  average  of  the  probabilities  that  two  different  non-infrastructure 
components  are  related  for  repair  purposes.  A  system  whose  index  mrk  is  small,  is 
considered  more  modular  that  one  whose  index  is  large.  The  modularity  index  muk  for 
technology  upgrades  is  defined  similarly. 

A  system  typically  has  a  projected  production  level,  targeted  to  provide  the  number 
of  units  needed  to  meet  the  need  for  which  the  system  is  being  developed.  This  projected 
production  level  can  change  over  time,  as  threats  change,  as  new  technologies/systems 
become  available,  or  as  costs  escalate.  Let  Pk  =  the  actual  production  level  for  system  k. 
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Acquisition  Enterprise  Model — Procurement 


The  acquisition  enterprise  consists  of  the  five  phases  of  a  defense  acquisition 
program,  as  defined  by  the  DoD  Defense  Acquisition  Guidebook  (DoD,  2006).  These 
phases  include  concept  development,  technology  development,  system  development, 
production  &  deployment,  and  operations  &  support.  Here,  the  focus  is  on  the  procurement 
phases  of  acquisition,  i.e.,  the  first  four  phases.  Operations  and  support  is  analogous  to 
sustainment  and  is  discussed  in  the  next  section.  These  phases  are  illustrated  in  Figure  2. 


•4 


Procurement 


>  Sustainment 


Figure  2.  Phases  of  Acquisition 

A  program  starts  in  the  first  phase  and  proceeds  through  the  remaining  phases. 
Sometimes,  this  process  involves  concurrency,  or  overlap  between  parts  of  two  different 
phases  (e.g.,  overlap  between  final  testing  in  system  development  and  low  rate  initial 
production).  In  the  procurement  phases  (the  first  four  phases  of  acquisition),  each  phase  is 
characterized  by  duration  and  a  cost  for  the  program.  Once  a  phase  is  concluded,  the 
program  moves  onto  the  next  phase.  The  model  assumes  no  concurrency.  This  basic 
model,  which  does  not  incorporate  modularity  or  production  level,  is  described  in  greater 
detail  by  Pennock  and  Rouse  (2008),  who  also  discuss  the  parameters  used  for  cost  and 
duration  figures  for  the  various  procurement  phases.  Elaborations  in  system  development 
and  in  production  are  discussed  below. 


■  Concept  development.  Concept  development  is  assumed  to  last  for  a  duration  of 
between  2  and  7.5  years,  with  a  mode  of  4.9.  Cost  is  assumed  to  be  a  linear 
function  of  duration,  with  rate  $20  million/year. 


■  Technology  development.  The  cost  and  duration  of  the  technology  development 
phase  depends  on  whether  the  program  is  using  evolutionary  or  traditional 
acquisition,  as  the  program  must  mature  the  technologies  to  be  used  for  the 
system  if  it  uses  traditional  acquisition.  Technologies  for  systems  being  acquired 
are  developed  exogenously  to  the  acquisition  enterprise,  from  the  military 
science  and  technology  (S&T)  enterprise.  These  technologies  are  then  brought 
into  a  program  when  needed  in  the  technology  development  phase.  The  maturity 
level  of  the  technologies  brought  into  a  program  is  used  to  characterize 
traditional  acquisition  versus  evolutionary  acquisition.  The  development  process 
is  risky,  in  that  individual  technologies  may  fail,  thus  causing  a  longer  technology 
development  phase  (and  higher  cost)  for  the  program.  A  program  that  uses 
immature  technologies  typically  has  a  longer,  costlier  and  riskier  technology 
development  phase  than  one  that  uses  relatively  more  mature  technologies.  The 
duration  and  cost  of  this  phase  is  the  time  and  cost  needed  to  develop  the 
needed  technologies  for  the  system. 


■  System  development.  Here,  we  assume  that  the  development  cost  and  time  are 
dependent  on  the  level  of  modularity.  In  particular,  they  are  increasing  functions 
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of  modularity.  We  assume  a  linear  relationship  between  cost  and  time.  For 
simplicity,  repair  and  upgrade  modularity  are  assumed  to  be  the  same  in  the 
model,  with  a  modularity  index  denoted  simply  as  mrk  =  muk  =  mk.  Recall  that  as 
mk  increases,  the  system  is  characterized  by  a  lesser  degree  of  modularity.  We 
then  let  the  system  development  time  Tdk  be  defined 


T  =  Ae 

1  dk  ^ 


Here,  A  is  a  scale  factor  related  to  the  time  in  years,  and  a  is  a  scale  factor 
that  represents  the  increasing  effort  it  takes  to  make  a  system  marginally 
more  modular.  In  the  model,  system  development  time  is  scaled  to  lie 
between  1.5  and  8  years,  and  its  cost  is  a  linear  function  of  duration  at  a  rate 
of  $1,000  million. 

■  Production  &  deployment.  The  cost  and  duration  of  the  production  phase  are, 
obviously,  dependent  on  the  production  level  of  the  system  in  question.  We  use 
the  Cobb-Douglas  production  function,  a  standard  micro-economic  model  that 
relates  input  units  to  units  produced  to  allow  for  increasing,  constant  or 
decreasing  returns  to  scale  (Kreps,  1990).  Letting  Xk  =  input  resources  used  for 
system  k,  Pk  =  production  of  k ,  and  b  =  scale  factor,  the  functional  form  is 


Inputs  here  are  capital,  labor  and  materials.  Letting  b  >  0  yields  increasing 
returns  to  scale,  which  is  typical  for  production  of  complex  systems. 
Assuming  a  constant  cost  for  inputs  Bk,  and  letting  Zk  =  the  cost  of  the 
production  phase,  we  obtain 


To  determine  the  mean  time  needed  for  the  production  phase  for  a  given 
production  level,  Zk  is  scaled  to  a  cost  between  $6,000  million  and  $18,800 
million,  and  the  production  time  is  determined  using  a  linear  relationship 
between  cost  and  duration  with  a  rate  of  $4,000  million/year. 


Acquisition  Enterprise  Model — Sustainment 

The  sustainment  model  includes  repairs  and  technology  upgrades  for  a  particular 
system  k.  The  production  level  Pk  is  assumed  to  be  the  fleet  size  for  the  duration  of 
sustainment  (i.e.,  no  systems  are  lost  or  retired).  Failures  and  technology  upgrades  are 
assumed  to  occur  randomly  according  to  a  Poisson  process.  Each  failure  or  technology 
upgrade  affects  only  one  component  directly.  Due  to  modularity,  though,  a  repair  to 
component  /  may  require  a  repair  to  another  component  j  (with  probability  rijk).  Similarly,  an 
upgrade  to  /  may  require  an  upgrade  to  another  component  j  (with  probability  uiJk).  The  rates 
for  the  failure  and  technology  upgrade  processes  are  defined  below. 

■  fj  is  the  failure  rate  associated  with  component  /. 

■  g,  is  the  repair  rate  associated  with  component  /. 

■  ti  is  the  arrival  rate  of  new  technology  upgrades  for  component  /. 

■  v,  is  the  upgrade  rate  for  component  /. 
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■  Pi  is  the  cost  of  repairing  component  /. 

■  q i  is  the  cost  associated  with  a  technology  upgrade  to  component  /. 

■  Cij  is  the  compatibility  cost  associated  with  making  component  j  technologically 
compatible  with  component  /  if  /  is  upgraded,  and  if  the  interaction  between  /  and  j 
necessitates  that  j  be  made  compatible  to  the  new  technology  for  /. 

Since  infrastructure  is  not  affected  by  repairs  or  technology  upgrades,  we  assume 
that  fi ,  gi,  U,  1/1,  pi,  <7i,  cn  and  cv  are  not  defined.  In  general,  it  is  assumed  that  ft  >  f„  g,  >  vh 
and  Pi  <  qh 

Once  a  system  is  deployed  into  sustainment,  it  experiences  failures  and  technology 
upgrades  for  its  various  components  according  to  the  above  rates.  Technology  upgrades 
are  a  significant  concern  for  systems  with  long  sustainment  lifecycles  (Singh  &  Sandborn, 
2006).  A  failure  invokes  a  repair  process,  and  a  technology  upgrade  opportunity  invokes  an 
upgrade  process.  Upon  occurrence  of  a  failure  in  component  /,  all  other  components  j  1 ) 
with  rjjk  >  0  are  evaluated  probabilistically,  using  random  number  generation,  to  determine  if 
a  repair  is  necessary  for  j.  Any  additional  components  are  then  repaired.  The  cost  is  the 
summation  of  the  repair  to  the  original  component  /  and  the  cost  of  other  components  being 
repaired.  Upon  occurrence  of  a  technology  upgrade  for  component  /',  all  other  components  j 
1)  with  Uijk  >  0  are  evaluated  probabilistically,  using  random  number  generation,  to 
determine  if  a  compatibility  upgrade  is  necessary  for  j.  Any  additional  components  are  then 
made  compatible.  The  cost  is  the  summation  of  the  summation  to  the  original  component  / 
and  the  cost  of  other  components  being  made  compatible.  If  a  failure  or  technology 
upgrade  for  /  arrives  while  the  system  is  in  downtime,  that  failure  or  technology  upgrade 
queues  until  the  downtime  is  resolved.  Multiple  entities  in  this  queue  are  processed  first- 
come-first-served.  This  process  continues  for  the  sustainment  life  of  the  system  over  the 
active  fleet,  which  is  assume  to  range  between  15  and  40  years,  with  a  mode  of  20  years  at 
a  median  rate  of  $965  million  per  year.  The  actual  rate  is  influenced  by  the  production  level. 

This  model  provides  cost  for  the  direct  activities  involving  maintenance  and  repair 
(including  upgrades)  within  the  sustainment  phases.  Clearly,  sustainment  encompasses 
many  other  costs.  Unger  (2009)  presents  analysis  of  the  sustainment  cost  categories  from 
the  DoD’s  Cost  Analysis  Improvement  Group  (CAIG)  for  Air  Force  programs.  Approximately 
52.5%  of  sustainment  cost  can  be  categorized  as  related  to  maintenance  and  repair.  We 
assume  that  approximately  half  of  these  costs  (i.e. ,  25%  of  sustainment  costs)  are  tied  to 
direct  repair  and  upgrade  activities  represented  in  our  sustainment  model. 

It  should  be  noted  that  there  is  no  effect  on  the  sustainment  phase  from  the 
acquisition  policy  used  in  the  procurement  phases  (i.e.,  traditional  versus  evolutionary). 

Exogenous  Model 

Exogenous  to  the  acquisition  enterprise  are  two  important  outside  influences.  The 
first  is  a  model  of  technical  progress,  which  represents  basic  research  performed  in  the 
commercial  world  or  via  non-defense  government  funding.  Results  from  this  model  are 
immature  technologies  that  are  input  into  the  second  exogenous  influence,  the  DoD  science 
and  technology  (S&T)  enterprise.  These  new  technologies  enter  the  S&T  enterprise  at  TRL 
1 .  Technologies  generated  by  the  model  of  technological  progress  increase  in  capability  as 
time  progresses  using  a  capability  growth  function  that  combines  a  learning  effect  (from 
other  DoD  applications)  and  an  exogenous  progress  effect  (from  commercial  and  outside 
technical  progress). 
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The  technology  development  process  model  matures  new  technologies  for  DoD 
systems,  typically  using  a  staged  process  of  development  whereby  ideas  are  reduced  to 
working  technologies  that  can  be  integrated  into  a  system.  There  is  considerable  technical 
risk  in  the  development  process,  as  ideas  often  do  not  work  in  practice,  do  not  scale  up  to 
production,  or  do  not  integrate  into  systems.  The  staged  process  mitigates  risk  by  not  fully 
funding  a  technology's  development,  allowing  it  to  be  culled  if  it  fails  or  if  it  is  outpaced  by 
competing  technologies.  It  should  be  noted  that  the  S&T  enterprise  model  consists  of  a 
single,  unified  organization,  rather  than  the  myriad  agencies  that  comprise  the  actual  DoD 
S&T  enterprise. 

Experiment 

This  section  describes  the  experiment  to  be  conducted. 

Parameters 

We  use  the  following  parameter  values  in  the  simulation  model  for  the  experiment: 

■  a  =  1.3 

■  b  =  1 .25 

■  1 1ti  ~  Triangular  (5,  8,  15)  years  for  all  /  and  over  all  systems 

■  Mfj  ~  Triangular  (1 .5,  2.5,  4)  years  for  all  /  and  over  all  systems 

■  pi  ~  uniform  (0.025,  0.075)  $  million  for  all  /  and  over  all  systems 

■  q i  ~  uniform  (0.2,  0.2375)  $  million  for  all  /  and  over  all  systems 

■  c,j  =  $0.1  million  for  all  /  and  j  and  over  all  systems 

■  k  =  6  over  all  systems 

It  should  be  noted  that  f(  and  f )  are  parameters  for  Poisson  processes,  and  that  their 
inverses  are  the  inter-arrival  times  for  those  respective  processes,  distributed  as  exponential 
variables.  Thus,  their  inter-arrival  times  are  shown  here.  Repair  times  and  upgrade  times 
are  assumed  to  be  instantaneous. 

Experimental  Design 

The  goal  is  to  study  the  effect  of  system  modularity  and  production  level  on 
acquisition  cost,  including  sustainment,  with  a  particular  focus  on  cost  of  evolutionary 
acquisition.  Thus,  we  adopt  a  factorial  experimental  design  with  three  independent 
variables,  as  outlined  in  Table  1. 


Table  1.  Experimental  Design 


Independent  Variable 

Level  1 

Level  2 

Acquisition  Policy 

[Traditional 

Evolutionary 

Modularity  Index 

p.50  (Low) 

P-25  (High) 

Production  Level 

^50  (Low) 

boo  (High) 
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This  results  in  a  23  factorial  experiment.  We  are  interested  in  studying  the  following 
six  dependent  variables: 

■  C-|  —  program  procurement  cost 

■  C2  =  program  sustainment  cost 

■  C3  =  total  program  cost  =  +  C2 

■  C4  =  annualized  procurement  cost  over  all  systems 

■  C5  =  annualized  sustainment  cost  over  all  systems 

■  C6  =  annualized  cost  over  all  systems  =  C4  +  C5 

Each  simulation  replication  is  run  for  a  period  of  150  years,  with  a  warm-up  period  of 
50  years  preceding.  This  allows  analysis  of  the  steady-state  enterprise  behavior,  since  the 
enterprise  model  needs  a  warm-up  period  to  reach  steady-state.  After  the  warm-up  period, 
the  statistics  collection  begins.  Ten  replications  of  each  combination  of  factors  are 
conducted  to  allow  for  statistical  significance. 

Experimental  Results 

Table  2  provides  summary  experimental  results.  Columns  two  through  four  contain 
the  level  of  each  independent  variable  as  shown  in  Table  1 .  The  value  shown  for  each 
dependent  variable  in  columns  five  through  ten  is  the  average  over  the  ten  replications.  The 
units  of  the  dependent  variables  are  in  millions  $. 


Table  1.  Summary  Experimental  Results 


Run 

Policy 

Mod. 

Prod. 

Ci 

c2 

c3 

c4 

c5 

c6 

1 

1 

1 

1 

12,657 

22,248 

34,906 

5,305 

5,075 

10,380 

2 

1 

1 

2 

18,640 

26,409 

45,049 

6,378 

5,653 

12,031 

3 

1 

2 

1 

13,832 

20,623 

34,455 

5,378 

4,441 

9,819 

4 

1 

2 

2 

20,078 

23,096 

43,174 

6,189 

4,296 

10,485 

5 

2 

1 

1 

11,518 

22,248 

33,766 

5,960 

6,316 

12,276 

6 

2 

1 

2 

17,548 

26,409 

43,957 

7,071 

6,556 

13,627 

7 

2 

2 

1 

12,629 

20,623 

33,252 

5,884 

5,210 

11,094 

8 

2 

2 

2 

18,910 

23,200 

42,109 

6,981 

5,290 

12,271 

Analysis  of  Overall  Results 

We  use  a  balanced  analysis  of  variance  (ANOVA)  method  to  determine  which 
independent  variables  (or  factors)  have  significant  effects  (Box,  Hunter  &  Hunter,  1978). 

The  ANOVA  also  computes  whether  there  are  significant  interaction  effects  among  more 
than  one  factor.  In  performing  the  analysis  of  variance  for  this  experiment,  Minitab®  version 
15  software  is  used.  Tables  3  through  8  report  the  analysis  of  variance  for  each  of  the 
dependent  variables,  Ci  through  C6,  respectively.  Main  effects  from  each  independent 
variable  are  noted,  as  are  interaction  effects  among  independent  variables. 
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Table  2.  Analysis  of  Variance  for  Program  Procurement  Cost  (C^ 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

26486575 

26486575 

384.31 

0 

Mod 

1 

32315116 

32315116 

468.88 

0 

Prod 

1 

752794480 

7.53E+08 

10922.85 

0 

Policy*Mod 

1 

24451 

24451 

0.35 

0.553 

Policy*Prod 

1 

8505 

8505 

0.12 

0.726 

Mod*Prod 

1 

330508 

330508 

4.8 

0.032 

Policy*Mod*Prod 

1 

207 

207 

0 

0.956 

Error 

72 

4962185 

68919 

Total 

79 

816922028 

From  the  analysis,  we  infer  that  each  of  the  main  effects  are  significant  (with  p  < 
0.10),  as  is  the  interaction  effect  between  modularity  and  production  level.  Similar  to 
Pennock  and  Rouse  (2008),  programs  using  evolutionary  acquisition  tend  to  have  a  lower 
program  cost  than  those  using  traditional  acquisition,  due  to  higher  technology  development 
costs.  As  expected,  low  levels  of  modularity  have  lower  procurement  costs,  due  to  less 
systems  engineering  work  in  the  development  phase.  Also  as  expected,  higher  production 
levels  lead  to  higher  procurement  costs.  High  modularity  and  high  production  level  interact 
to  increase  procurement  cost  more  than  each  individual  factor. 


Table  3.  Analysis  of  Variance  for  Program  Sustainment  Cost  (C2) 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

13394 

13394 

0.01 

0.933 

Mod 

1 

119354720 

119354720 

63.02 

0 

Prod 

1 

223440510 

223440510 

117.97 

0 

Policy*Mod 

1 

13394 

13394 

0.01 

0.933 

Policy*Prod 

1 

13394 

13394 

0.01 

0.933 

Mod*Prod 

1 

13382709 

13382709 

7.07 

0.01 

Policy*Mod*Prod 

1 

13394 

13394 

0.01 

0.933 

Error 

72 

136366569 

1893980 

Total 

79 

492598085 

For  sustainment  cost,  there  is  no  effect  from  acquisition  policy.  This  is  to  be 
expected,  since  the  simulation  model  assumes  no  difference  in  the  sustainment  profile 
between  the  two  policies.  However,  the  effects  from  modularity,  production  level  and  their 
interaction  are  significant.  Low  levels  of  modularity  tend  to  cause  higher  sustainment  costs, 
and  high  production  levels,  of  course,  result  in  higher  sustainment  costs.  Introducing  high 
modularity  into  a  system  tends  to  mitigate  the  sustainment  cost  increase  associated  with 
high  production  levels. 
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Table  4.  Analysis  of  Variance  for  Total  Program  Cost  (C3) 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

25308713 

25308713 

12.6 

0.001 

Mod 

1 

27460953 

27460953 

13.67 

0 

Prod 

1 

1796490516 

1796490516 

894.42 

0 

Policy*Mod 

1 

1651 

1651 

0 

0.977 

Policy*Prod 

1 

43247 

43247 

0.02 

0.884 

Mod*Prod 

1 

9506987 

9506987 

4.73 

0.033 

Policy*Mod*Prod 

1 

10271 

10271 

0.01 

0.943 

Error 

72 

144615531 

2008549 

Total 

79 

2003437868 

Looking  at  total  program  cost,  each  of  the  main  effects  is  significant,  as  is  the 
interaction  between  modularity  and  production  level.  The  effect  of  the  main  factors  is 
explained  by  the  combined  effect  of  these  factors  on  the  constituents  of  C3  (i.e. ,  Ci  and  C2). 
However,  the  effect  of  modularity  combines  opposite  effects  noted  in  C i  and  C2.  The  effect 
of  reduced  cost  from  increased  modularity  noted  in  sustainment  cost  wins  out,  as 
sustainment  costs  overpower  development  costs.  Similarly,  the  interaction  effects  between 
modularity  and  production  level  in  each  of  the  constituents  are  in  opposite  directions.  Once 
again,  the  effect  from  sustainment  costs  wins  out,  and  we  can  infer  that  high  modularity 
helps  mitigate  the  effect  of  cost  increases  due  to  high  production  levels. 


Table  5.  Analysis  of  Variance  for  Annualized  Procurement  Cost  (C4) 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

8745812 

8745812 

853.98 

0 

Mod 

1 

99301 

99301 

9.7 

0.003 

Prod 

1 

20921804 

20921804 

2042.9 

0 

Policy*Mod 

1 

3027 

3027 

0.3 

0.588 

Policy*Prod 

1 

131703 

131703 

12.86 

0.001 

Mod*Prod 

1 

94557 

94557 

9.23 

0.003 

Policy*Mod*Prod 

1 

78033 

78033 

7.62 

0.007 

Error 

72 

737368 

10241 

Total 

79 

30811604 

In  analyzing  annualized  procurement  cost,  we  find  that  each  of  the  main  effects  is 
significant,  as  are  most  of  the  interaction  effects.  Confirming  Pennock  and  Rouse  (2008), 
the  annualized  procurement  cost  for  evolutionary  acquisition  is  significantly  higher  than  that 
of  traditional  acquisition.  This  is  due  to  the  higher  number  of  refresh  procurement  cycles. 
Similar  to  C-i,  high  production  levels  are  associated  with  higher  annualized  costs.  However, 
in  this  case,  low  levels  of  modularity  are  associated  with  higher  annualized  costs,  in  contrast 
to  the  effect  from  C-i.  This  is  likely  due  to  the  increased  development  time  required  for  high 
levels  of  modularity,  which  reduces  the  number  of  systems  deployed  over  the  lifecycle  and, 
hence,  the  annualized  cost.  Likewise,  there  is  a  corresponding  significant  interaction  effect 
between  modularity  and  production  levels  whereby  low  levels  of  modularity  in  conjunction 
with  high  production  levels  are  associated  with  increased  costs.  Interestingly,  there  is  a 
significant  interaction  effect  here  between  acquisition  policy  and  production  level.  The 
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increased  number  of  programs  under  evolutionary  acquisition  interacts  with  high  production 
levels  to  increase  annualized  procurement  costs  more  than  each  individual  factor.  Finally, 
there  is  a  three-way  interaction  effect  among  all  independent  variables.  This  is  manifested 
as  a  reduction  in  the  difference  in  cost  between  different  production  levels  and  acquisition 
policies  as  modularity  level  is  increased. 


Table  6.  Analysis  of  Variance  for  Annualized  Sustainment  Cost  (C5) 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

19079848 

19079848 

134.12 

0 

Mod 

1 

23789768 

23789768 

167.23 

0 

Prod 

1 

707084 

707084 

4.97 

0.029 

Policy*Mod 

1 

181790 

181790 

1.28 

0.262 

Policy*Prod 

1 

16048 

16048 

0.11 

0.738 

Mod*Prod 

1 

971557 

971557 

6.83 

0.011 

Policy*Mod*Prod 

1 

397276 

397276 

2.79 

0.099 

Error 

72 

10242758 

142261 

Total 

79 

55386129 

For  annualized  sustainment  cost,  the  three  main  effects  are  again  significant.  For 
modularity  and  production  level,  these  effects  are  consistent  with  the  reasons  noted  for 
program  sustainment  cost  C2.  For  acquisition  policy,  the  effect  is  consistent  with  the 
analysis  for  annualized  procurement  cost  C4,  i.e.,  the  increased  number  of  refresh  cycles 
means  that  there  is  an  increased  number  of  programs  needing  sustainment.  In  this  model, 
we  assume  that  the  program  sustainment  profiles  are  the  same  under  evolutionary  and 
traditional  acquisition.  This  does  not  account  for  the  possibility  that,  under  an  evolutionary 
system,  it  may  be  the  case  that  system  lifecycles  are  reduced,  allowing  a  reduction  in 
sustainment  costs.  The  interaction  effect  between  modularity  and  production  level  is 
consistent  with  the  effect  noted  for  individual  program  sustainment  cost  C2,  whereby  high 
modularity  mitigates  cost  increases  associated  with  high  levels  of  production. 


Table  7.  Analysis  of  Variance  for  Annualized  Total  Cost  (C6) 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

53661197 

53661197 

292.72 

0 

Mod 

1 

26963050 

26963050 

147.08 

0 

Prod 

1 

29321345 

29321345 

159.95 

0 

Policy*Mod 

1 

231729 

231729 

1.26 

0.265 

Policy*Prod 

1 

55804 

55804 

0.3 

0.583 

Mod*Prod 

1 

1672308 

1672308 

9.12 

0.003 

Policy*Mod*Prod 

1 

827448 

827448 

4.51 

0.037 

Error 

72 

13199104 

183321 

Total 

79 

125931985 

Finally,  in  terms  of  annualized  total  cost,  each  of  the  three  main  effects  is  significant, 
as  is  the  interaction  effect  between  modularity  and  production  level.  The  effect  for 
acquisition  policy  is  explained  by  the  effect  noted  for  the  constituents  of  C6  (i.e.,  C4  and  C5), 
i.e.,  the  increased  number  of  programs  due  to  evolutionary  acquisition.  The  effect  for 
modularity  is  explained  by  the  larger  effect  of  modularity  in  reducing  sustainment  costs  than 
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increasing  procurement  costs,  while  the  effect  of  production  level  is  explained  simply  by  the 
increased  cost  of  producing  and  sustaining  more  units. 

Analysis  of  Evolutionary  Acquisition 

We  now  focus  on  evolutionary  acquisition  by  analyzing  the  experimental  results  that 
just  pertain  to  it.  The  observations  below  summarize  our  findings  relative  to  those  of  the 
overall  experiment.  These  results  come  from  reducing  the  observations  to  a  22  factorial 
experiment  involving  only  modularity  and  production  level  as  independent  variables. 

■  For  Ci,  the  program  procurement  cost,  both  modularity  and  production  level  have 
similar  significant  effects.  The  interaction  effect  is  somewhat  weaker,  though, 
registering  only  a  p-value  of  0.136.  Thus,  we  infer  that  this  interaction  effect 
predominates  for  traditional  acquisition  due  to  the  relatively  larger  program 
procurement  cost. 

■  Similarly,  for  C2,  the  annualized  sustainment  cost,  both  modularity  and  production 
level  have  the  same  type  of  significant  effect.  The  interaction  effect  is  also 
somewhat  weaker  than  in  the  overall  experiment,  with  a  p-value  of  0.077. 

■  The  same  observations  hold  for  C3.  Here,  for  the  effect  of  modularity  and  the 
interaction  effect  between  modularity  and  production  level,  the  sustainment  cost 
predominates,  causing  increased  modularity  to  have  a  reducing  effect  on  the 
total  program  cost.  The  p-value  for  the  interaction  effect  is  relatively  weak  at 
0.138. 

■  For  the  annualized  costs  (C4,  C5  and  C6),  modularity  retains  its  significant  effect 
in  the  same  direction  as  in  the  overall  experiment.  However,  production  level  is 
significant  only  for  C4  and  C6.  In  addition,  the  interaction  effect  between 
modularity  and  production  is  not  significant  across  the  three  dependent  variables. 
This  is  likely  captured  in  the  three-way  interaction  effect  among  the  three 
independent  variables  noted  in  the  overall  experiment  for  C4,  C5  and  C6.  It  may 
be  that  additional  replications  are  needed  to  get  statistically  significant  results  for 
these  effects.  This  bears  further  investigation. 


Discussion  and  Future  Research 


The  above  results  imply  a  number  of  conclusions  relevant  for  military  acquisition. 

■  Increased  system  modularity  yields  increases  in  the  system  development  cost, 
but  the  decrease  in  sustainment  cost  over  the  system  lifecycle  may  more  than 
compensate  for  these  increased  costs.  This  points  to  the  need  to  view 
acquisition  as  an  investment  process.  While  the  short-term  budgeting  nature  of 
the  federal  government  works  against  this  perspective,  a  longer  term  view  does 
show  the  benefit  of  investing  current  costs  to  achieve  long-term  savings. 

■  Modularity  can  help  mitigate  the  cost  increases  associated  with  higher  production 
levels  through  an  interaction  effect  between  these  two  factors.  Similar  to  the 
previous  point,  this  effect  involves  the  way  in  which  sustainment  costs  overpower 
those  of  development,  due  to  long  lifecycles. 

■  Evolutionary  acquisition  seems  less  susceptible,  especially  from  an  annualized 
cost  point  of  view,  to  this  interaction  effect  between  modularity  and  production 
levels.  While  this  bears  further  investigation,  it  should  be  noted  that  those 
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programs  that  do  maintain  characteristics  of  traditional  acquisition  may  wish  to 
investigate  this  phenomenon. 

■  Evolutionary  acquisition  may  decrease  individual  program  costs,  but  the  more 
frequent  refresh  cycles  may  drive  cost  growth  in  overall  procurement  and 
sustainment.  Thus,  discretion  is  needed  in  managing  these  refresh  cycles, 
especially  when  high  production  levels  are  involved. 

This  work  has  addressed  the  process  aspects  of  the  acquisition  enterprise.  Clearly, 
processes  are  an  important  and  critical  part  of  the  acquisition  enterprise.  However, 
acquisition  occurs  in  the  context  of  organizational  behavior  that  is  impacted  by  incentives 
and  information  availability.  The  DoD  has  spent  significant  resources  on  incentive  programs 
to  facilitate  positive  acquisition  outcomes.  Some  research  suggests  that  these  resources 
have  not  been  used  effectively  (GAO,  2005).  However,  economic  research  suggests  that  it 
is  possible  to  design  incentive  programs  under  different  information  availability  scenarios 
(Hildebrandt,  2009).  Thus,  an  avenue  of  future  research  is  to  integrate  organizational 
behavior  modeling  of  acquisition,  combined  with  process  modeling. 
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related  to  one  other,  or  connected 
to  one  another 

-  In  a  non-modular  system, 
changes  to  one  component  may 
necessitate  changes  to  another  (if 
they  are  related) 

-  Modularity  is  usually 
conceptualized  as  a  matrix 
showing  the  relationships  between 
components 

-  Infrastructure  may  exist 
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•  Repair  matrix  Rk 

•  Component  rjJk  =  probability 
that  repair  to  component  / 
causes  repair  to  component  j 

•  Similar  matrix  for  upgrades 
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Hypotheses 
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•  Research  literature 

-  Increased  modularity  decreases  the  cost  of  implementing 
technology  upgrades  for  deployed  systems 

-  Increased  modularity  decreases  the  mean  time  to  repair  a  failed 
system  component  and  potentially  the  cost  for  the  repair 

-  Increased  modularity  increases  systems  engineering  cost 
associated  with  system  development. 

•  This  research 

-  What  is  the  trade-off  between  increased  development  cost 
versus  decreased  sustainment  cost? 

-  What  is  the  impact  on  this  trade-off  of  the  system  production 
level? 
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Model  Overview 
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•  Discrete-event  simulation  model  implemented  in  ARENA 

•  Three  acquisition  sub-models 

-  System  model  -  the  system  being  acquired 

-  Procurement  model  -  the  processes  through  which  a  system  is 
procured 

-  Sustainment  model  -  the  processes  through  which  a  system  is 
sustained 

•  Exogenous  model  -  external  organizations  and  effects 

-  DoD  S&T  enterprise 

-  External  technology  progress 
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System  Model 
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Technologies 

-  Application  area 

-  Maturity 

-  Capability 
Modularity 

-  Extent  to  which  system 
components  affect  one 
another 

-  Repair  and  technology 
upgrades 

-  Modularity  index 
Production  level 


Procurement  Model 
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Concept  development 

Technology  development 

-  Develop  technologies  within  a 
program  vs.  accept  only 
mature  technologies 

System  development 

-  Exponentially  increasing  effort 
(cost  and  time)  needed  to 
make  a  system  more  modular 

Production  &  deployment 

-  Cobb-Douglas  production 
function  with  increasing 
returns  to  scale  based  on 
production  level 
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Sustainment  Model 
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•  Repair  and  technology  upgrade 
processes 

•  Poisson  arrival  process  assumed 

-  Component  failure  rate 

-  Upgrade  opportunity  rate 

•  Modularity  impacts  whether  a 
repair/upgrade  to  component  / 
causes  one  to  component  j 

•  No  effect  from  mission  profile  on 
arrival  rates 

•  Costs  based  on  individual 
transactions  over  fleet 

•  Costs  are  scaled  to  be  25%  of 
sustainment  budget 

•  Fleet  is  assumed  to  equal 
production  level 
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Experimental  Design  Ge^Js"“ 


•  Independent  variables 

-  Acquisition  policy  (traditional  vs.  evolutionary) 

-  Modularity  index  (0.50  vs.  0.25) 

-  Production  level  (250  vs.  500) 

•  Dependent  variables 

-  Program  procurement  cost 

-  Program  sustainment  cost 

-  Total  program  cost 

-  Annualized  procurement  cost  (over  all  systems) 

-  Annualized  sustainment  cost  (over  all  systems) 

-  Annualized  total  cost  (over  all  systems) 

•  23  factorial  experiment  for  each  dependent  variable 
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Simulation  Details 
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•  Ten  replications  of  each  factor  combination  were 
conducted  for  statistical  significance 

•  Each  replication  consist  of  150  years  of  simulated  time, 
with  a  prior  50  year  warm-up  period  to  reach  steady 
state 
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Summary  Results 
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Run 

Pol. 

Mod. 

Prod. 

p-p 

P-S 

P-T 

A-P 

A-S 

A-T 

1 

T 

0.50 

250 

12,657 

22,248 

34,906 

5,305 

5,075 

10,380 

2 

T 

0.50 

500 

18,640 

26,409 

45,049 

6,378 

5,653 

12,031 

3 

T 

0.25 

250 

13,832 

20,623 

34,455 

5,378 

4,441 

9,819 

4 

T 

0.25 

500 

20,078 

23,096 

43,174 

6,189 

4,296 

10,485 

5 

E 

0.50 

250 

11,518 

22,248 

33,766 

5,960 

6,316 

12,276 

6 

E 

0.50 

500 

17,548 

26,409 

43,957 

7,071 

6,556 

13,627 

7 

E 

0.25 

250 

12,629 

20,623 

33,252 

5,884 

5,210 

11,094 

8 

E 

0.25 

500 

18,910 

23,200 

42,109 

6,981 

5,290 

12,271 

P-P:  Program  procurement  cost 
P-S:  Program  sustainment  cost 
P-T:  Total  program  cost 


A-P:  Annualized  procurement  cost 
A-S:  Annualized  sustainment  cost 
A-T:  Total  annualized  cost 


Cost  figures  in 
millions  $ 
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Analysis 
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•  Statistical  analysis  is  performed  using  balanced  analysis 
of  variance  (ANOVA)  method 

•  This  allows  detection  of  significant  effects  on  dependent 
variables  from 

-  Independent  variable  factors  (e.g.,  acquisition  policy) 

-  Interactions  between  factors 
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Program  Cost 
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•  Procurement 

-  Evolutionary  acquisition  tends  to  lower  procurement  cost,  due  to  lower 
technology  development  costs. 

-  High  levels  of  modularity  tend  to  escalate  procurement  cost,  due  to 
higher  systems  engineering  costs. 

-  High  modularity  and  high  production  levels  interact  to  increase 
procurement  more  than  single  factor  effects. 

•  Sustainment 

-  Low  modularity  tends  to  increase  sustainment  cost  due  to  increased 
repairs  and  upgrades. 

-  High  modularity  mitigates  the  increased  sustainment  cost  associated 
with  high  production  levels. 

•  Overall 

-  High  modularity  tends  to  lower  overall  acquisition  cost  and  mitigates  the 
overall  cost  associated  with  high  production. 


r 
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Annualized  Enterprise  Cost  Ge^|sraffii 


•  Procurement 

-  Evolutionary  acquisition  tends  to  increase  procurement  cost,  due  to 
increased  refresh  cycles. 

-  Low  levels  of  modularity  tend  to  increase  procurement  cost.  High 
modularity  lengthens  development  times,  resulting  in  fewer  systems 
being  deployed. 

-  Low  modularity  and  high  production  interact  to  increase  cost. 

-  Increased  number  of  programs  under  evolutionary  acquisition  interacts 
significantly  with  high  production  to  increase  annualized  cost. 

•  Sustainment 

-  High  modularity  results  in  reduced  sustainment  cost. 

-  An  increased  number  of  programs  under  evolutionary  acquisition  leads 
to  higher  sustainment  costs. 

-  High  modularity  mitigates  the  increased  cost  of  high  production  levels. 

•  Overall 

-  High  modularity  tends  to  lower  overall  acquisition  cost  and  mitigates  the 
overall  cost  associated  with  high  production. 
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Implications  for  EA 
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•  Based  on  consideration  of  results  for  evolutionary 
acquisition  only  (22  experiment). 

•  Modularity  had  similar  effects  under  evolutionary 
acquisition  as  in  the  overall  experiment. 

•  The  interaction  effect  between  modularity  and  production 
level  is  weaker  for  evolutionary  acquisition  than  for 
traditional. 

•  This  implies  that  the  interaction  effect  is  experienced 
mainly  in  traditional  acquisition. 
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Conclusions 


Georgia 


•  Increases  in  system  modularity  can  result  in  overall  cost 
reduction,  as  cost  decreases  in  sustainment  overpower 
cost  increases  in  development.  This  points  to  the 
importance  of  upstream  investments. 

•  Modularity  can  help  mitigate  cost  increases  associated 
with  higher  production  levels. 

•  Evolutionary  acquisition  is  less  susceptible  to  this  effect, 
especially  from  an  annualized  cost  perspective. 

•  Evolutionary  acquisition  may  decrease  individual 
program  costs,  but  can  increase  overall  annualized  cost. 
Thus,  discretion  should  be  used  in  refresh  cycles. 
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Program  Procurement  Ge^h|s"“ 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

26486575 

26486575 

384.31 

0 

Mod 

1 

32315116 

32315116 

468.88 

0 

Prod 

1 

752794480 

752794480 

10922.85 

0 

Policy*Mod 

1 

24451 

24451 

0.35 

0,553 

Policy*Prod 

1 

8505 

8505 

0.12 

0.726 

Mod*Prod 

1 

330508 

330508 

4.8 

0.032 

Policy*Mod*Prod 

1 

207 

207 

0 

0.956 

Error 

72 

4962185 

68919 

Total 

79 

816922028 
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Program  Sustainment  °^isssr“ 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

13394 

13394 

0.01 

0.933 

Mod 

1 

119354720 

119354720 

63.02 

0 

Prod 

1 

223440510 

223440510 

117.97 

0 

Policy*Mod 

1 

13394 

13394 

0.01 

0.933 

Policy*Prod 

1 

13394 

13394 

0.01 

0.933 

Mod*Prod 

1 

13382709 

13382709 

7.07 

0.01 

Policy*Mod*Prod 

1 

13394 

13394 

0.01 

0.933 

Error 

72 

136366569 

1893980 

Total 

79 

492598085 
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Total  Program  Cost 
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Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

25308713 

25308713 

12.6 

0,001 

Mod 

1 

27460953 

27460953 

13.67 

0 

Prod 

1 

1796490516 

1796490516 

894.42 

0 

Policy*Mod 

1 

1651 

1651 

0 

0.977 

Policy*Prod 

1 

43247 

43247 

0.02 

0.884 

Mod*Prod 

1 

9506987 

9506987 

4.73 

0.033 

Policy*Mod*Prod 

1 

10271 

10271 

0.01 

0.943 

Error 

72 

144615531 

2008549 

Total 

79 

2003437868 
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Annualized  Procurement 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

8745812 

8745812 

853.98 

0 

Mod 

1 

99301 

99301 

9.7 

0,003 

Prod 

1 

20921804 

20921804 

2042.9 

0 

Policy*Mod 

1 

3027 

3027 

0.3 

0.588 

Policy*Prod 

1 

131703 

131703 

12.86 

0.001 

Mod*Prod 

1 

94557 

94557 

9.23 

0,003 

Policy*Mod*Prod 

1 

78033 

78033 

7.62 

0,007 

Error 

72 

737368 

10241 

Total 

79 

30811604 
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Annualized  Sustainment 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

19079848 

19079848 

134.12 

0 

Mod 

1 

23789768 

23789768 

167.23 

0 

Prod 

1 

707084 

707084 

4.97 

0.029 

Policy*Mod 

1 

181790 

181790 

1.28 

0.262 

Policy*Prod 

1 

16048 

16048 

0.11 

0.738 

Mod*Prod 

1 

971557 

971557 

6.83 

0.011 

Policy*Mod*Prod 

1 

397276 

397276 

2.79 

0.099 

Error 

72 

10242758 

142261 

Total 

79 

55386129 
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Total  Annualized  Cost  Ge^|»" 


Source 

DF 

SS 

MS 

F 

P 

Policy 

1 

53661197 

53661197 

292.72 

0 

Mod 

1 

26963050 

26963050 

147.08 

0 

Prod 

1 

29321345 

29321345 

159.95 

0 

Policy*Mod 

1 

231729 

231729 

1.26 

0,265 

Policy*Prod 

1 

55804 

55804 

0.3 

0.583 

Mod*Prod 

1 

1672308 

1672308 

9.12 

0.003 

Policy*Mod*Prod 

1 

827448 

827448 

4.51 

0,037 

Error 

72 

13199104 

183321 

Total 

79 

125931985 
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